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(54) Image enhancement on JPEG compressed image data 



(57) A method and apparatus are provided for en- 
hancing JPEG compressed digital image data wherein 
only the DC components of the compressed image data 
are required to obtain a statistical subsampling of the 
corresponding uncompressed image data. JPEG com- 
pressed image data are received that represent an input 
digital image, and the JPEG compressed image data in- 
clude a plurality of DC components. DC components are 
extracted from the JPEG compressed image data, and 
the extracted plurality of said DC components represent 
a subsampling of the input digital image. At least some 
of the plurality of extracted DC components are input to 



an automated image enhancement system. The DC 
components input to the automated image enhance- 
ment system are used to derive at least one of a final 
correction tone reproduction curve and a sharpness fil- 
ter for enhancement of the input digital image represent- 
ed by the JPEG compressed data. The final correction 
tone reproduction curve and/or sharpness filter are 
bound to the JPEG compressed image data for subse- 
quent use in enhancing the input digital image repre- 
sented by the JPEG compressed image data after the 
JPEG compressed image data have been decom- 
pressed. 
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Description 

[0001] In the past, a typical application for copiers or 
scan-to-print image processing systems was to repro- 
duce an input image as accurately as possible, i.e., 
render a copy. Thus, copies have been rendered as ac- 
curately as possible, including flaws in the source im- 
age. However, as customers have become more so- 
phisticated in their document reproduction require- 
ments, they have recognized that an exact copy is often 
not desired. Instead, an inexact copy that is perceived 
as a higher quality image has been deemed more de- 
sirable. 

[0002] Until recently, image quality from the output of 
a copier or a scan-to-print system was directly related 
to the quality of the input image. While this has been 
satisfactory for reproducing images of text or line draw- 
ings, it has been found to be sub-optimal for purposes 
of reproducing photographs and other more complex 
images. With photographs, in particular, reproduction is 
complicated given the inexact nature of the discipline, 
variations in equipment, aging of photographs, and the 
like. Given that the input image is often poor quality, it 
has been deemed desirable to render an output image 
that is perceived to be the "best" possible image (or at 
least superior to the input image), rather than an exact 
copy. 

[0003] U.S. Patents to Eschbach et al. and Fuss et al. 
disclose an Automated Image Enhancement (AIE) sys- 
tem. This system receives an optionally sub-sampled 
description of the input image, and alters the tone repro- 
duction curve (TRC) -- a curve that defines the relation- 
ship of an input image signal to an output image signal 
for purposes of enhancement -- and/or derives or alters 
a sharpness filter for that image on an image-by-image 
basis, as appropriate, so that the resulting output image 
is perceived to be superior to the input image. AIE is 
used, for example, to alter perceived exposure, lumi- 
nance, sharpness, saturation, color balance, and the 
like to provide an output image that is preferable to the 
input image. It is important to note that AIE performs well 
even if it has to rely upon a sub-sampled image to sta- 
tistically analyze the overall image. AIE does not require 
that each and every item of image information be ana- 
lyzed. Of course, this subsampling speeds image en- 
hancement operations and reduces the size/number of 
memory buffers required to implement the AIE system. 
[0004] More particularly, for example, U.S. Patent No. 
5,414,538 entitled "Image-Dependent Exposure En- 
hancement" discloses a method of altering the per- 
ceived exposure of an output image produced from an 
input image that includes: (a) receiving the input image 
defined in terms of red-green-blue (RGB) signals; (b) 
converting the RGB signals to corresponding lumi- 
nance-chrominance signals including at least one signal 
that represent overall image intensity; (c) comparing the 
intensity signal to upper and lower intensity threshold 
signals that define the acceptable levels of brightness 



and darkness in the image; (d) if one of the thresholds 
is exceeded, the image signal representative of image 
intensity is processed according to a select equation, 
and a TRC associated with the image is adjusted so that 

5 exposure characteristics of the resulting output image 
are perceived to be superior to those of the input image. 
[0005] U.S. Patent No. 5,450,502 entitled "Image-De- 
pendent Luminance Enhancement" discloses a method 
of altering the perceived luminance of an output image 

10 produced from an input image that includes: <a) receiv- 
ing the input image data defined in terms of a color 
space; (b) if required, converting the input image data 
into a luminance-chrominance color space wherein at 
least one term bears a relationship to overall intensity 

15 of the input image; (c) deriving a global intensity histo- 
gram for the overall input image; <d) filtering the histo- 
gram signal to flatten high peaks and low valleys without 
altering its relatively flat portions; and, (e) utilizing the 
filtered histogram signal to control TRC mapping in a 

20 device with which the image is to be rendered. In ac- 
cordance with another aspect of the disclosed lumi- 
nance enhancement method, the input image can be di- 
vided into plural regions, and a local intensity histogram 
signal can be derived for each region. If any of the local 

25 histogram signals are flatter than the global histogram 
signal, the local signals are summed and used in place 
of the global histogram as input to the histogram flatten- 
ing filter. 

[0006] The methods described in the Eschbach et al. 

30 '538 and *502 patents, and the other Eschbach et al. and 
Fuss et al. patents noted above, are described in con- 
nection with uncompressed image data. However, in 
many image processing operations, the image data is 
retrieved from an image storage device or other location 

35 or is otherwise supplied in a compressed form to mini- 
mize image storage space. Most commonly, the image 
data is compressed according to the Joint Photographic 
Expert Group (JPEG) recommendation ISO DIS 
10918-1 that has become an international standard for 

40 lossy compression of still images. 

[0007] Thus, heretofore, image enhancement opera- 
tions according to the AIE systems described above 
have required that the JPEG (or otherwise) compressed 
images first be decompressed or "decoded" for image 

45 enhancement operations. This is undesirable in that the 
decompression operation slows the overall enhance- 
ment operation, the image processing apparatus must 
be provided with additional memory to accommodate 
large amounts of uncompressed image data, and each 

so JPEG or other lossy compression operation, itself, fur- 
ther degrades the image data due to data loss. 
[0008] In light of the foregoing, it has been deemed 
desirable to provide a novel and non-obvious method 
for utilizing JPEG compressed image data in the image 

55 enhancement operations described in the above-noted 
Eschbach et al. and Fuss et al. AIE patents. This will 
allow JPEG compressed images to be enhanced for 
subsequent rendering without the image being decom- 
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pressed, enhanced, and then recompressed. 
[0009] In accordance with the present invention, a 
method and apparatus for image enhancement of JPEG 
compressed image data are provided. In accordance 
with a first aspect of the present invention, a method of 
processing JPEG compressed image data comprises: 
(i) receiving JPEG compressed image data that repre- 
sent an input digital image, the JPEG compressed im- 
age data including a plurality of DC components; (ii) ex- 
tracting a plurality of the DC components from the JPEG 
compressed image data; (iii) inputting at least some of 
the extracted DC components to an automated image 
enhancement system; (iv) using the DC components in- 
put to the automated image enhancement system to de- 
rive a final correction tone reproduction curve and/or a 
sharpness filter for enhancement of the input digital im- 
age represented by the JPEG compressed data; (v) de- 
compressing the JPEG compressed image data to ob- 
tain decompressed image data that represent the input 
digital image ; and, (vi) using the final tone reproduction 
curve and/or the sharpness filter to enhance the decom- 
pressed image data. 

[0010] One advantage of the present invention re- 
sides in the provision of a method and apparatus for en- 
hancing JPEG compressed image data, without fully de- 
compressing the data. 

FIGURE 1 is a block diagram illustrating image en- 
hancement of JPEG compressed image data in ac- 
cordance with the present invention; 
FIGURE 2 is a partial illustration of an input digital 
image to be compressed according to a convention- 
al JPEG data compression operation; 
FIGURE 3 is a diagrammatic illustration of an 8x8 
pixel block of JPEG compressed data, and extrac- 
tion of the DC component therefrom for purposes 
of image enhancement in accordance with the 
present invention; and, 

FIGURE 4 illustrates a image enhancement appa- 
ratus and method for enhancement of JPEG com- 
pressed data in accordance with the present inven- 
tion. 

[0011] FIGURE 1 illustrates an example of an image 
processing system implementing the present invention. 
Input image data defining an input image is supplied 
from either a scanner 10a or memory 10b. In the case 
where the input image data is supplied from a scanner 
10a, the scanner can be a black-and-white or color 
scanner that scans a printed input image and that de- 
rives digital image data that defines the scanned printed 
image. Typically, the scanner 10a outputs digital image 
data defined in terms of red, green, and blue (RGB) color 
separations, although the data can alternatively be sup- 
plied in any other suitable color space. A color space 
converter 11 receives the digital image data from the 
scanner 10a and, if required, converts the data into a 
luminance-chrominance color space, such as YC b C r 



4 

color space or the like as is required for JPEG data com- 
pression of the input image data. The input image data 
is output by the color space converter 1 1 to a J P EG com- 
pression unit 12 that compresses the data using a con- 

5 ventional JPEG compression process. 

[0012] Alternatively, the input image data is supplied 
from a memory 10b or other conventional source in 
JPEG compressed format as is well known to minimize 
memory requirements. In either case, whether the input 

10 image data is supplied from the scanner 1 0a, the mem- 
ory 10b, or another suitable source, the data can define 
the input image monochromatically or in terms of multi- 
ple color separations that, together, define a color im- 
age. 

15 [0013] As is described in full detail below, JPEG com- 
pressed data includes DC components and, in accord- 
ance with the present invention, these DC components 
are extracted from the JPEG compressed data by a DC 
component extractor 14 that receives the JPEG com- 
20 pressed data from either the JPEG compression unit 12 
or another source such as the memory 10b. The DC 
components extracted by the DC component extractor 
14 are input to the Automated Image Enhancement 
(AIE) system 16 that is fully described in the aforemen- 
25 tioned Eschbach et al. and Fuss et al. patents. The un- 
altered JPEG compressed image data, including the DC 
components, is also input from the JPEG compression 
unit 12 or the memory 10b to a TRC and Filter Binder 
18. It should be noted that optionally one may choose 
30 to incorporate a modified DC component extractor as 
described in U.S. Statutory Invention Registration No. 
H0001684. In such case, a small subset of the low fre- 
quency AC components is also decoded in an efficient 
way. In this case, some of the performance advantages 
35 of the pure DC component extraction with respect to 
processing time are traded off against an increased set 
of data for the statistical analysis. This trade-off can be 
advantageous, e.g., for images with a limited number of 
sampling points. For simplicity of description, it is intend- 
40 ed that the terms "DC" and "DC components" used in 
the following encompass this optional implementation. 
[0014] The AIE system 16, in turn, derives a correc- 
tion tone reproduction curve TRC and/or filter based up- 
on only the DC components input from the DC compo- 
45 nent extractor 14. The correction TRC and/or filter are 
input to the TRC & filter binder 18 wherein the TRC and 
filter are bound to or otherwise operatively associated 
with the JPEG compressed image data. Once the cor- 
rection TRC and/or filter are bound to or otherwise op- 
50 eratively associated with the relevant JPEG com- 
pressed image data, the correction TRC and filter are 
available to any downstream image processing module, 
and the TRC and filter can be applied once the JPEG 
compressed image data are decompressed. For exam- 
55 pie, when the input image is to be viewed by way of an 
image output terminal 22, such as a printer or visual dis- 
play, the JPEG compressed data, including the correc- 
tion TRC and/or filter bound thereto, are input to a de- 
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compress and enhance unit 20 wherein: (i) the JPEG 
compressed data are decompressed in a conventional 
manner to provide uncompressed input image data; 
and, (ii) the uncompressed input image data are en- 
hanced via application of the corrective TRC and/or fil- 
ter. Thereafter, the enhanced, uncompressed input im- 
age data are input to the image output terminal for dis- 
play/printing. 

[0015] According to the essential properties of JPEG 
compression, an input image I (FIGURE 2) is defined in 
terms of a plurality of pixels P, each defined in terms of 
a luminance-chrominance color space such as YC b C r . 
This image I to be compressed is divided into a 2-dimen- 
sional array of typically square blocks B of pixels P (the 
individual pixels P in only one of the blocks B are labeled 
for clarity). Most commonly, the original image I is divid- 
ed into square blocks with each block comprising 8x8 = 
64 pixels P from the original image I. 
[0016] A discrete cosine transform (DCT) is then per- 
formed on the pixel data P in each block B. The forward 
DCT has the effect of transforming each of the blocks B 
into the spatial frequency domain and, following the 
DCT, the elements in a block B still completely describe 
the original input image data, but larger values tend to 
cluster at the top left comer of the block B, in a low spa- 
tial frequency region. Simultaneously, the elements lo- 
cated more toward the lower right hand portion of the 
block B will tend toward zero for most photographic im- 
ages. The top-left entry 30 in each block B, which rep- 
resents the average value all pixels P in the block B, is 
known as the DC component or DC coefficient (identi- 
fied in FIGURE 3 as "DC") of the block B, and all the 
other entries 32 in the block are referred to as the AC 
coefficients or AC components (identified in FIGURE 3 
as "AC") of the block. In the case of color image data, 
the DC component DC of a block B represents the av- 
erage luminance-chrominance value of the pixels P de- 
fining that block B. Of course, in the case of black-and- 
white data, the DC component DC of a block B repre- 
sents only the average luminance of the pixels P defin- 
ing that block B. 

[0017] Following the DCT step, individual entries DC, 
AC in the block B are quantized, or in effect made into 
smaller numbers, and rounded. Then, the quantized en- 
tries are Huffman-encoded to yield a string of bits. There 
may be other lossless compression steps to encode the 
quantized DCT coefficients, but the final product is in- 
evitably a string of bits for each block B, wherein each 
block B is converted into a string of bits of a different 
length. Of course, numerous variations on the above- 
described JPEG compression technique are well known 
and may be implemented without departing from the 
overall scope and intent of the present invention. 
[0018) FIGURE 3 illustrates operation of the DC com- 
ponent extractor 14 as previously described in relation 
to FIGURE 1 . In FIGURE 3 it may be seen that when a 
block B of JPEG compressed data is input to the DC 
component extractor 14, the DC component extractor 
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14 identifies and outputs the DC component DC of the 
input block B. As noted, this extracted DC component 
DC represents the average luminance or the average 
luminance-chrominance value forthe pixels P of a block 
s B. 

[0019] FIGURE 4 is a more detailed illustration of an 
image processing apparatus that is adapted for en- 
hancement of JPEG compressed image data in accord- 
ance with the present invention. JPEG compressed im- 
10 age data is received at 50 from a memory 10b, a JPEG 
compression unit 12, or another source, and the DC 
component extractor 14 extracts at least some, and 
preferably all, of the DC components DC from the JPEG 
compressed data. Those of ordinary skill in the art will 
15 recognize that all of the DC components DC need not 
necessarily be extracted to provide a good subsampling 
of the JPEG compressed data » only some of the DC 
components DC are required to obtain a usable statis- 
tical sampling of the JP EG compressed image data. The 
20 JPEG compressed data stream is unaltered by the DC 
component extractor 14, and from the DC component 
extractor 14, it is input to the TRC/Filter binder unit 18 
by way of a data channel 100. 
[0020] The extracted DC components DC are input 
25 via data channel 1 02 to a DC component control module 
104 that controls the input of the extracted DC compo- 
nents DC into the automated image enhancement (AIE) 
unit 16. In particular, the DC component control unit 104 
makes the extracted DC components DC available on 
30 an information channel 108 that is, in turn, operatively 
connected to a plurality of AIE modules, e.g., an AIE 
exposure module 112, an AIE color balance module 
122, and AIE contrast module 132, and AIE luminance 
module 1 82, and an AIE sharpness module 1 92, by way 
35 of buses 110, 120, 130, 180, 190, respectively. Those 
of ordinary skill in the art will recognize that other AIE 
modules can form a part of the AIE unit 16, and it is not 
intended that the invention be limited to the particular 
AIE modules shown. The AIE modules can also return 
40 information to the information channel 108 for use by 
other AIE modules. The AIE modules 112, 122, 132, 
182, 192, in the AIE unit 16 are described in the afore- 
mentioned Eschbach et al. and Fuss et al. U.S. Patents 
in greater detail. 
45 [0021] Each of the exposure, color balance, contrast, 
and luminance AIE modules 112, 122, 132, 182 use the 
data supplied on the information channel 108 to calcu- 
late the input signals for the respective TRC generators 
114, 124, 134, 184 that generate intermediate TRC's re- 
so quired to effect the required image modifications as de- 
termined by each module 112, 122, 132, 182. Each of 
the TRC generators 114, 124, 134, 184, directs a gen- 
erated intermediate TRC to a TRC combiner 140 that 
combines the respective intermediate TRC's into a sin- 
55 gie, final TRC 1 60 defined by a look-up table (LUT). The 
final TRC is then directed to the bind TRC unit 150 of 
the TRC/filter binder 18 wherein ft is bound to the JPEG 
compressed image data for subsequent use. Similarly, 
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a sharpness module 192 receives the DC components 
DC from the information channel 108 and uses same to 
generate a sharpness filter 194 that is directed to the 
TRC/filter binder 18, in particular to a bind filter unit 200 
thereof, wherein the fitter is bound to the JPEG com- 
pressed data (along with any TRC that has already been 
bound to the JPEG compressed data by the bind TRC 
unit 150. The combined TRC/Filter and JPEG com- 
pressed image data is output at 250 to downstream im- 
age processing modules such as the decompress/en- 
hance unit 20 and image output terminal 22 described 
in connection with FIGURE 1. It should be recognized 
by those of ordinary skill in the art that the final TRC 1 60 
and/or sharpness filter 194 can be bound or otherwise 
operatively associated with the JPEG compressed im- 
age data for any desired length of time, including a frac- 
tion of a second or months, years, or longer such as for 
long-term archival storage. Furthermore, the final TRC 
1 60 and/or filter 1 94 can be used immediately once de- 
rived, assuming the JPEG compressed image data has 
been suitably decompressed. 

[0022] Those of ordinary skill in the art will recognize 
that the DC components DC extracted from the JPEG 
compressed data provide a subsampling of the uncom- 
pressed input image data, without requiring that the im- 
age data be further decompressed. Thus, the DC com- 
ponents can be used by the AIE systems described 
above without significant modifications to these sys- 
tems. On the other hand, it is necessary to keep in mind 
that the DC components DC, themselves, represent on- 
ly an average value of the data in each 64 pixel (or other 
size) block B of the input image I. Therefore, it may be 
desirable in certain situations to use more conservative 
parameters in the AIE system 16 to account for the fact 
that any given pixel P in a block B may vary significantly 
from the average value DC. Also, as noted above, it is 
to be recognized that not all the extracted DC compo- 
nents DC need to be input to the AIE system 16. For 
example, half of the DC components DC may provide a 
good subsampling of the JPEG compressed data. 



Claims 

1. A method of processing JPEG compressed image 
data comprising: 



10 



15 2. 



20 



25 3. 



30 



35 



40 



enhancement system; 

using said DC components input to said auto- 
mated image enhancement system to derive at 
least one of a final correction tone reproduction 
curve and a sharpness filter for enhancement 
of said input digital image represented by said 
JPEG compressed data; and, 
decompressing said JPEG compressed image 
data to obtain decompressed image data that 
represent said input digital image; and, 
using said at least one of said final tone repro- 
duction curve and said sharpness filter to en- 
hance said decompressed image data. 

The method of processing JPEG compressed im- 
age data as set forth in claim 1 , further comprising, 
prior to said step of decompressing said J P EG com- 
pressed image data, the step of: 

at least temporarily storing said at least one 
of said final tone reproduction curve and said sharp- 
ness filter for subsequent use in enhancing said in- 
put digital image represented by said JPEG com- 
pressed image data. 

The method of processing JPEG compressed im- 
age data as set forth in claim 1 , wherein said step 
of inputting at least some of said plurality of extract- 
ed DC components to an automated image en- 
hancement system comprises inputting at least 
some of said plurality of DC components to at least 
one of an exposure enhancement module, a color 
balance enhancement module, a contrast enhance- 
ment module, and a luminance enhancement mod- 
ule, said at least one module receiving and deriving 
a respective intermediate correction tone reproduc- 
tion curve based upon at least said input DC com- 
ponents, wherein said step of deriving at least one 
of a final correction tone reproduction curve and a 
sharpness filter comprises inputting each respec- 
tive intermediate tone reproduction curve to a tone 
reproduction curve combiner and combining all of 
said respective intermediate tone reproduction 
curves to define said final correction tone reproduc- 
tion curve . 
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receiving JPEG compressed image data that 
represent an input digital image, said JPEG 
compressed image data including a plurality of 
DC components; 

extracting a plurality of said DC components 
from said JPEG compressed image data, said 
extracted plurality of said DC components rep- 
resenting a subsampling of said input digital im- 
age; 

inputting at least some of said plurality of ex- 
tracted DC components to an automated image 
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(54) Image enhancement on JPEG compressed image data 



(57) A method and apparatus are provided for en- 
hancing JPEG compressed digital image data wherein 
only the DC components of the compressed image data 
are required to obtain a statistical subsampling of the 
corresponding uncompressed image data. JPEG com- 
pressed image data are received that represent an input 
digital image, and the JPEG compressed image data in- 
clude a plurality of DC components. DC components are 
extracted from the JPEG compressed image data, and 
the extracted plurality of said DC components represent 
a subsampling of the input digital image. At least some 
of the plurality of extracted DC components are input to 



an automated image enhancement system. The DC 
components input to the automated image enhance- 
ment system are used to derive at least one of a final 
correction tone reproduction curve and a sharpness fil- 
ter for enhancement of the input digital image represent- 
ed by the JPEG compressed data. The final correction 
tone reproduction curve and/or sharpness filter are 
bound to the JPEG compressed image data for subse- 
quent use in enhancing the input digital image repre- 
sented by the JPEG compressed image data after the 
JPEG compressed image data have been decom- 
pressed. 
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